Skip to main content

Measurement: Project-Based Data Sufficiency & Disqualification Guidelines

Overview

This document includes guidelines that define the minimum data requirements for accurate modeling and measurement of project-based programs (e.g., energy efficiency, electrification). The guidelines are rooted in the sufficiency requirements for the Recurve platform’s project-based OpenDSM hourly, daily, and billing models, with additional elements that arise in measuring real-world programs. Meters that do not meet these standards may be assigned a savings estimate based on the measured population and/or may be excluded from reporting to maintain analytical integrity.

Consumption Data Definitions

The consumption data types listed throughout the document are defined below for context.

Monthly:

  • Monthly data typically provides 1 value per meter for each month’s total usage or each billing cycle’s total usage
  • Default logic rolls down monthly data to daily data by taking the monthly or billing total value and dividing evenly across days in each month
  • Default logic checks data sufficiency at the daily level
  • Electric data must have >328 days of non-zero consumption data

Daily:

  • Daily data typically provides 1 value per meter for each day’s total usage
  • Default logic checks data sufficiency at the daily level
  • Electric data must have >328 days of non-zero consumption data

Hourly:

  • Hourly data typically provides 1 value per meter for each hour’s total usage, or at a more granular level (e.g., 15 minutes, 5 minutes, etc.)
  • Default logic rolls up more granular consumption data to hourly values
  • Default logic checks data sufficiency at the hourly level

Training Period Requirements

Data Sufficiency Requirements​

The training period must include approximately 1 year of data before the start of each project’s blackout period. The following criteria apply to the training period (the 365-day period before the program intervention) for meter, temperature, and solar irradiance data when a solar model is used (i.e., for hourly measurements), as well as to the overlap across all sources.

At least 329 days within the 1-year training period must contain data for 90% or more of expected readings. The number of days of consumption and temperature data missing should not exceed 36 days (10%) for billing and daily methods:

  • Monthly: Meters must have a minimum of 11 months with consumption data
  • Daily: Meters must have at least 90% of days (> 328 days) with consumption data
  • Hourly: Meters must have at least:
    • 90% of all training period days
    • at least 90% of the expected data
    • And, 90% of the hours within each calendar month’s expected data

Meters that fail these requirements will be disqualified, and the reason will be documented.

DER Installation Date Requirements​

Meters that have solar or battery installation dates during the training period will be disqualified from measurement, and the reason will be documented. Exceptions are made for installation dates that fall within a grace period (default: 30 days) at the beginning of the training period.

Model Fit Requirements​

Meters must have consumption patterns that allow for the generation of a model with a reasonable fit.

Recurve’s platform at the start of each project’s performance period computes a model’s:

  • Coefficient of Variation of Root Mean Squared Error (CVRMSE),
  • And its Percentile Normalized Root Mean Squared Error (PNRMSE), defined as the RMSE divided by the interquartile range of a meter’s interval consumption

These metrics are used to disqualify meters that exceed certain thresholds according to our OpenDSM model’s default criteria.

CVRMSE or PNRMSE is calculated at the start of each project’s performance period, once full training data is available:

  • Hourly: OpenDSM hourly model CVRMSE between 0 and 1.4 OR PNRMSE < 2.2
  • Daily/Billing: OpenDSM daily/billing model CVRMSE between 0 and 1.0 OR PNRMSE < 1.6

Meters that fail these thresholds will be disqualified due to poor model fit.

Reporting Period Requirements

Data Sufficiency Requirements​

The reporting period must include at least 1 year of data following the end of the project blackout period. The following criteria are applied to the reporting period (the 365-day period after the blackout end date or beyond) and are analogous to those established for the training period noted above. However, this criterion must be assessed on an ongoing basis as the first reporting year progresses.

At least 329 days within the 1-year reporting period must contain data for 90% or more of expected readings. The number of days of consumption and temperature data missing should not exceed 36 days (10%) for billing and daily methods:

  • Monthly: Meters must have a minimum of 11 months with consumption data
  • Daily: Meters must have at least 90% of days (> 328 days) with consumption data
  • Hourly: Meters must have at least 90% of hours within each calendar month with consumption data

Treatment meters that fail these reporting period thresholds will be disqualified, and the reason will be documented.

DER Installation Date Requirements​

Meters that have solar or battery installation dates during the first year of the reporting period will be disqualified from measurement, and the reason will be documented. Exceptions are made for installation dates that fall within a grace period (default: 30 days) at the end of the first reporting year.

Comparison Pool Requirements​

When comparison group correction is used, each treatment meter must have a sufficiently large pool of non-participant comparison-group meters that meet all training period requirements and measurement period data sufficiency requirements. Treatment meters for which a suitable comparison group cannot be constructed will be disqualified, and the reason will be documented.

Savings Outlier Thresholds​

Very high or very low savings relative to the counterfactual often indicate that a building’s consumption has changed in ways not associated with a program. Configurable savings outlier thresholds provide a systematic opportunity for review based on expected savings ranges.

Default Energy Efficiency savings outlier thresholds:

  • Reporting year 1 savings (savings/counterfactual) should be within the bounds of -50% to 50%
  • Outlier thresholds can and should be revisited based on statistical analysis of the program being measured or similar past programs

Meters that fail these thresholds will be disqualified, but may be further reviewed and overridden for qualified or disqualified status.

Other Requirements

Dual enrollment disqualification​

To prevent duplicative incentives and entanglement in measured program impacts, new projects are screened during the enrollment phase for prior participation in other programs. The default dual enrollment logic checks whether the submitted project has participated in another program within the past year. If the new enrollment date falls within the last 12 months of the previous program's enrollment, the project will either not be allowed to enter Recurve's program or be disqualified. Recurve checks for dual enrollment based on all of the client’s programs in the Recurve App database, and based on past participation data provided by the client

Disqualification Treatments

Meters that do not meet data sufficiency requirements or are disqualified for reasons such as poor model fit will still receive a performance value based on the program-level configuration. In these cases, a meter may be assigned a value aligned with the measured performance of participating meters in the same measurement grouping.

For more details on Recurve's approach to disqualification and savings assignment, see this document.